Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study.

OBJECTIVES: To determine the reproducibility of biomedical systematic review search strategies. STUDY DESIGN AND SETTING: A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling six key Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension (PRISMA-S) reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records. RESULTS: The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; six searches differed by more than 1,000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible. CONCLUSION: Systematic review search reporting is poor. To correct this will require a multifaceted response from authors, peer reviewers, journal editors, and database providers.

Open peer review urgently requires evidence: A call to action.

Open Peer Review is gaining prominence in attention and use, but to responsibly open up peer review, there is an urgent need for additional evidence. Here, we propose a preliminary research agenda and issue a call to action.

What do Retraction Notices Reveal About Institutional Investigations into Allegations Underlying Retractions?

Academic journal publications may be retracted following institutional investigations that confirm allegations of research misconduct. Retraction notices can provide insight into the role institutional investigations play in the decision to retract a publication. Through a content analysis of 7,318 retraction notices published between 1927 and 2019 and indexed by the Web of Science, we found that most retraction notices (73.7%) provided no information about institutional investigations that may have led to retractions. A minority of the retraction notices (26.3%) mentioned an institutional investigation either by journal authorities (12.1%), research performing organizations (10.3%), joint institutions (1.9%), research integrity and ethics governing bodies (1.0%), third-party institutions (0.5%), unspecified institutions (0.4%), or research funding organizations (0.1%). Comparing retraction notices issued before and after the introduction of retraction guidelines by the Committee on Publication Ethics (COPE) in 2009 revealed that those published after the guidelines' publication were more likely to report investigations by journal authorities. Comparing retraction notices from different disciplines revealed that those from social sciences and the humanities were more likely to disclose investigations by research performing organizations than those from biomedical and natural sciences. Based on these findings, we suggest that the COPE retraction guidelines in the future make it mandatory to disclose in retraction notices institutional investigations leading to retractions.

Co-creating Research Integrity Education Guidelines for Research Institutions.

To foster research integrity (RI), research institutions should develop a continuous RI education approach, addressing various target groups. To support institutions to achieve this, we developed RI education guidelines together with RI experts and research administrators, exploring similarities and differences in recommendations across target groups, as well as recommendations about RI education using approaches other than formal RI training. We used an iterative co-creative process. We conducted four half-day online co-creation workshops with 16 participants in total, which were informed by the RI education evidence-base. In the first two workshops, participants generated ideas for guidelines' content, focusing on different target groups and various approaches to RI education. Based on this content we developed first drafts of the guidelines. Participants in the third and fourth workshop refined those drafts. We next organized a working group which further prioritized, reorganized, and optimized the content of the guidelines. We developed four guidelines on RI education focusing on (a) bachelor, master and PhD students; (b) post-doctorate and senior researchers; (c) other RI stakeholders; as well as (d) continuous RI education. Across guidelines, we recommend mandatory RI training; follow-up refresher training; informal discussions about RI; appropriate rewards and incentives for active participation in RI education; and evaluation of RI educational events. Our work provides experience-based co-created guidance to research institutions on what to consider when developing a successful RI education strategy. Each guideline is offered as a distinct, publicly available tool in our toolbox ( www.sops4ri.eu/toolbox ) which institutions can access, adapt and implement to meet their institution-specific RI education needs.Trial registration https://osf.io/zej5b .

The Contribution of Moral Case Deliberation to Teaching RCR to PhD Students.

Teaching responsible conduct of research (RCR) to PhD students is crucial for fostering responsible research practice. In this paper, we show how the use of Moral Case Deliberation-a case reflection method used in the Amsterdam UMC RCR PhD course-is particularity valuable to address three goals of RCR education: (1) making students aware of, and internalize, RCR principles and values, (2) supporting reflection on good conduct in personal daily practice, and (3) developing students' dialogical attitude and skills so that they can deliberate on RCR issues when they arise. What makes this method relevant for RCR education is the focus on values and personal motivations, the structured reflection on real experiences and dilemmas and the cultivation of participants' dialogical skills. During these structured conversations, students reflect on the personal motives that drive them to adhere to the principles of good science, thereby building connections between those principles and their personal values and motives. Moreover, by exploring personal questions and dilemmas related to RCR, they learn how to address these with colleagues and supervisors. The reflection on personal experiences with RCR issues and questions combined with the study of relevant normative frameworks, support students to act responsibly and to pursue RCR in their day-to-day research practice in spite of difficulties and external constraints.

Research misconduct and questionable research practices form a continuum.

Research data mismanagement (RDMM) is a serious threat to accountability, reproducibility, and re-use of data. In a recent article in this journal, it was argued that RDMM can take two forms: intentional research misconduct or unintentional questionable research practice (QRP). I disagree because the scale for severity of consequences of research misbehavior is not bimodal. Furthermore, intentionality is difficult to prove beyond doubt and is only one of many criteria that should be taken into account when deciding on the severity of a breach of research integrity and whether a sanction is justified. Making a distinction between RDMM that is research misconduct and RDMM which not puts too much emphasis on intentionality and sanctioning. The focus should rather be on improving data management practices by preventive actions, in which research institutions should take a leading role.

Transparency in conducting and reporting research: A survey of authors, reviewers, and editors across scholarly disciplines.

Calls have been made for improving transparency in conducting and reporting research, improving work climates, and preventing detrimental research practices. To assess attitudes and practices regarding these topics, we sent a survey to authors, reviewers, and editors. We received 3,659 (4.9%) responses out of 74,749 delivered emails. We found no significant differences between authors', reviewers', and editors' attitudes towards transparency in conducting and reporting research, or towards their perceptions of work climates. Undeserved authorship was perceived by all groups as the most prevalent detrimental research practice, while fabrication, falsification, plagiarism, and not citing prior relevant research, were seen as more prevalent by editors than authors or reviewers. Overall, 20% of respondents admitted sacrificing the quality of their publications for quantity, and 14% reported that funders interfered in their study design or reporting. While survey respondents came from 126 different countries, due to the survey's overall low response rate our results might not necessarily be generalizable. Nevertheless, results indicate that greater involvement of all stakeholders is needed to align actual practices with current recommendations.

Why research integrity matters and how it can be improved.

Scholars need to be able to trust each other, because otherwise they cannot collaborate and use each other's findings. Similarly trust is essential for research to be applied for individuals, society or the natural environment. The trustworthiness is threatened when researchers engage in questionable research practices or worse. By adopting open science practices, research becomes transparent and accountable. Only then it is possible to verify whether trust in research findings is justified. The magnitude of the issue is substantial with a prevalence of four percent for both fabrication and falsification, and more than 50% for questionable research practices. This implies that researchers regularly engage in behaviors that harm the validity and trustworthiness of their work. What is good for the quality and reliability of research is not always good for a scholarly career. Navigating this dilemma depends on how virtuous the researcher at issue is, but also on the local research climate and the perverse incentives in the way the research system functions. Research institutes, funding agencies and scholarly journals can do a lot to foster research integrity, first and foremost by improving the quality of peer review and reforming researcher assessment.

How to combine rules and commitment in fostering research integrity?

Research integrity (RI) is crucial for trustworthy research. Rules are important in setting RI standards and improving research practice, but they can lead to increased bureaucracy; without commensurate commitment amongst researchers toward RI, they are unlikely to improve research practices. In this paper, we explore how to combine rules and commitment in fostering RI. Research institutions can govern RI using markets (using incentives), bureaucracies (using rules), and network processes (through commitment and agreements). Based on Habermas' Theory of Communicative Action, we argue that network processes, as part of the lifeworld, can legitimize systems - that is, market or bureaucratic governance modes. This can regulate and support RI practices in an efficient way. Systems can also become dominant and repress consensus processes. Fostering RI requires a balance between network, market and bureaucratic governance modes. We analyze the institutional response to a serious RI case to illustrate how network processes can be combined with bureaucratic rules. Specifically, we analyze how the Science Committee established at Tilburg University in 2012 has navigated different governance modes, resulting in a normatively grounded and efficient approach to fostering RI. Based on this case, we formulate recommendations to research institutions on how to combine rules and commitment.

In Defense of the Netherlands Code of Conduct for Research Integrity: Response to Radder.

We assess Radder's criticisms of the Netherlands Code of Conduct for Research Integrity and show that they either miss their mark or depend on controversial background assumptions about the purpose of the Code. Although Radder raises important questions about the broader roles and purposes of research in society, his conclusion that the Code should be revised in the ways he proposes is unjustified.

Indicators of questionable research practices were identified in 163,129 randomized controlled trials.

OBJECTIVES: To explore indicators of the following questionable research practices (QRPs) in randomized controlled trials (RCTs): (1) risk of bias in four domains (random sequence generation, allocation concealment, blinding of participants and personnel, and blinding of outcome assessment); (2) modifications in primary outcomes that were registered in trial registration records (proxy for selective reporting bias); (3) ratio of the achieved to planned sample sizes; and (4) statistical discrepancy. STUDY DESIGN AND SETTING: Full texts of all human RCTs published in PubMed in 1996-2017 were automatically identified and information was collected automatically. Potential indicators of QRPs included author-specific, publication-specific, and journal-specific characteristics. Beta, logistic, and linear regression models were used to identify associations between these potential indicators and QRPs. RESULTS: We included 163,129 RCT publications. The median probability of bias assessed using Robot Reviewer software ranged between 43% and 63% for the four risk of bias domains. A more recent publication year, trial registration, mentioning of CONsolidated Standards Of Reporting Trials-checklist, and a higher journal impact factor were consistently associated with a lower risk of QRPs. CONCLUSION: This comprehensive analysis provides an insight into indicators of QRPs. Researchers should be aware that certain characteristics of the author team and publication are associated with a higher risk of QRPs.

Using co-creation methods for research integrity guideline development - how, what, why and when?

Existing research integrity (RI) guideline development methods are limited in including various perspectives. While co-creation methods could help to address this, there is little information available to researchers and practitioners on how, why and when to use co-creation for developing RI guidelines, nor what the outcomes of co-creation methods are. In this paper, we aim to address this gap. First, we discuss how co-creation methods can be used for RI guideline development, based on our experience of developing RI guidelines. We elaborate on steps including preparation of the aims and design; participant sensitization; organizing and facilitating workshops; and analyzing data and translating them into guidelines. Secondly, we present the resulting RI guidelines, to show what the outcome of co-creation methods are. Thirdly, we reflect on why and when researchers might want to use co-creation methods for developing RI guidelines. We discuss that stakeholder engagement and inclusion of diverse perspectives are key strengths of co-creation methods. We also reflect that co-creation methods have the potential to make guidelines implementable if followed by additional steps such as revision working groups. We conclude that co-creation methods are a valuable approach to creating new RI guidelines when used together with additional methods.

Replication and trustworthiness.

This paper explores various relations that exist between replication and trustworthiness. After defining "trust", "trustworthiness", "replicability", "replication study", and "successful replication", we consider, respectively, how trustworthiness relates to each of the three main kinds of replication: reproductions, direct replications, and conceptual replications. Subsequently, we explore how trustworthiness relates to the intentionality of a replication. After that, we discuss whether the trustworthiness of research findings depends merely on evidential considerations or also on what is at stake. We conclude by adding replication to the other issues that should be considered in assessing the trustworthiness of research findings: (1) the likelihood of the findings before the primary study was done (that is, the prior probability of the findings), (2) the study size and the methodological quality of the primary study, (3) the number of replications that were performed and the quality and consistency of their aggregated findings, and (4) what is at stake.

Superb supervision: A pilot study on training supervisors to convey responsible research practices onto their PhD candidates.

One way to strengthen research integrity, is through supervision. According to previous research, a supervisor should be well-versed in responsible research practices (RRPs) and possess the necessary interpersonal skills to convey RRPs. We developed a 3-day pilot training for PhD supervisors that combined RRPs and interpersonal skills. Our aim was to assess: perceptions regarding supervision skills (before and after the pilot) and participants' views on combining RRPs and interpersonal skills. Before and after the pilot, we sent the Research Supervision Quality Evaluation survey to the participating PhD supervisors and their PhD candidates. The pilot was concluded with a focus group where participants deliberated over the combination of training in interpersonal skills and RRPs and whether such training should become compulsory. Both supervisors and PhD candidates were more positive about the supervisor's interpersonal skills and the ability to foster RRPs after the training. Participants were enthusiastic about the training's dual focus but believed that making the training compulsory would be undesirable. The results highlight the potential of RRPs training for supervisors. However, caution is warranted, as the results regard a small sample of volunteering supervisors, underscoring the need for larger programs to foster responsible supervision that are rigorously evaluated.

Research funders should be more transparent: a plea for open applications.

Transparency is increasingly becoming the new norm and modus operandi of the global research enterprise. In this mini-review, we summarize ongoing initiatives to increase transparency in science and funding in particular. Based on this, we make a plea for the next step in funders' compliance with the principles of Open Science, suggesting the adoption of open applications. Our proposed model includes a plea for the publication of all submitted grant applications; open sharing of review reports, argumentations for funding decisions and project evaluation reports; and the disclosure of reviewers' and decision committee members' identities. In line with previous calls for transparency and the available evidence about these measures' effectiveness, we argue that open applications could lead to more diverse collaboration, recognition of research ideas, fairer procedures for grant allocation, more research on funding practices and increased trust in the funding allocation process.

Prevalence of responsible research practices among academics in The Netherlands.

Background: Traditionally, research integrity studies have focused on research misbehaviors and their explanations. Over time, attention has shifted towards preventing questionable research practices and promoting responsible ones. However, data on the prevalence of responsible research practices, especially open methods, open codes and open data and their underlying associative factors, remains scarce. Methods: We conducted a web-based anonymized questionnaire, targeting all academic researchers working at or affiliated to a university or university medical center in The Netherlands, to investigate the prevalence and potential explanatory factors of 11 responsible research practices. Results: A total of 6,813 academics completed the survey, the results of which show that prevalence of responsible practices differs substantially across disciplines and ranks, with 99 percent avoiding plagiarism in their work but less than 50 percent pre-registering a research protocol. Arts and humanities scholars as well as PhD candidates and junior researchers engaged less often in responsible research practices. Publication pressure negatively affected responsible practices, while mentoring, scientific norms subscription and funding pressure stimulated them. Conclusions: Understanding the prevalence of responsible research practices across disciplines and ranks, as well as their associated explanatory factors, can help to systematically address disciplinary- and academic rank-specific obstacles, and thereby facilitate responsible conduct of research.

Promoting trust in research and researchers: How open science and research integrity are intertwined.

Proponents of open science often refer to issues pertaining to research integrity and vice versa. In this commentary, we argue that concepts such as responsible research practices, transparency, and open science are connected to one another, but that they each have a different focus. We argue that responsible research practices focus more on the rigorous conduct of research, transparency focuses predominantly on the complete reporting of research, and open science's core focus is mostly about dissemination of research. Doing justice to these concepts requires action from researchers and research institutions to make research with integrity possible, easy, normative, and rewarding. For each of these levels from the Center for Open Science pyramid of behaviour change, we provide suggestions on what researchers and research institutions can do to promote a culture of research integrity. We close with a brief reflection on initiatives by other research communities and stakeholders and make a call to those working in the fields of research integrity and open science to pay closer attention to one other's work.

Stakeholders' Experiences of Research Integrity Support in Universities: A Qualitative Study in Three European Countries.

Fostering research integrity (RI) increasingly focuses on normative guidance and supportive measures within institutions. To be successful, the implementation of support should be informed by stakeholders' experiences of RI support. This study aims to explore experiences of RI support in Dutch, Spanish and Croatian universities. In total, 59 stakeholders (Netherlands n = 25, Spain n = 17, Croatia n = 17) participated in 16 focus groups in three European countries. Global themes on RI support experiences were identified by thematic analysis. Themes identified were: 'RI governance and institutional implementation', 'RI roles and structures', 'RI education and supervision', and 'Infrastructure, technology and tools supporting daily practice'. Experiences of support differed between countries in relation to: the efforts to translate norms into practice; the extent to which RI oversight was a responsibility of RE structures, or separate RI structures; and the availability of support close to research practice, such as training, responsible supervision, and adequate tools and infrastructure. The study reinforces the importance of a whole institutional approach to RI, embedded within local jurisdictions, rules, and practices. A whole institutional approach puts the emphasis of responsibility on institutions rather than individual researchers. When such an approach is lacking, some stakeholders look for intervention by authorities, such as funders, outside of the university.

Designing and implementing a research integrity promotion plan: Recommendations for research funders.

Various stakeholders in science have put research integrity high on their agenda. Among them, research funders are prominently placed to foster research integrity by requiring that the organizations and individual researchers they support make an explicit commitment to research integrity. Moreover, funders need to adopt appropriate research integrity practices themselves. To facilitate this, we recommend that funders develop and implement a Research Integrity Promotion Plan (RIPP). This Consensus View offers a range of examples of how funders are already promoting research integrity, distills 6 core topics that funders should cover in a RIPP, and provides guidelines on how to develop and implement a RIPP. We believe that the 6 core topics we put forward will guide funders towards strengthening research integrity policy in their organization and guide the researchers and research organizations they fund.